Which configuration options are preferred in spark?

Question

I wanted to enquire that which configuration option is given priority in spark? Is it the configuration file or the options we manually specify when running the spark-submit

Answer 1

The Spark (1.5.0) configuration page clearly states what the priorities are:

Properties set directly on the SparkConf take highest precedence, then flags passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.

So this is the priority order (from highest to lowest):

• Properties set on the SparkConf (in program).
• Flags passed to spark-submit or spark-shell.
• Options set in the spark-defaults.conf file.