PySpark SparkSession Builder with Kubernetes Master

Question

I recently saw a pull request that was merged to the Apache/Spark repository that apparently adds initial Python bindings for PySpark on K8s. I posted a comment to the PR asking

Answer 1

pyspark client mode works on Spark's latest version 2.4.0

This is how I did it (in Jupyter lab):

import os
os.environ['PYSPARK_PYTHON']="/usr/bin/python3.6"
os.environ['PYSPARK_DRIVER_PYTHON']="/usr/bin/python3.6"

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

sparkConf = SparkConf()
sparkConf.setMaster("k8s://https://localhost:6443")
sparkConf.setAppName("KUBERNETES-IS-AWESOME")
sparkConf.set("spark.kubernetes.container.image", "robot108/spark-py:latest")
sparkConf.set("spark.kubernetes.namespace", "playground")

spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
sc = spark.sparkContext

Note: I am running kubernetes locally on Mac with Docker Desktop.