What is the difference between spark-submit and pyspark?

Question

If I start up pyspark and then run this command:

import my_script; spark = my_script.Sparker(sc); spark.collapse(\'./data/\')

Everything is

Answer 1

1. If you built a spark application, you need to use spark-submit to run the application

   • The code can be written either in python/scala

   • The mode can be either local/cluster

2. If you just want to test/run few individual commands, you can use the shell provided by spark

   • pyspark (for spark in python)
   • spark-shell (for spark in scala)