Enable case sensitivity for spark.sql globally

问题

The option spark.sql.caseSensitive controls whether column names etc should be case sensitive or not. It can be set e.g. by

spark_session.sql('set spark.sql.caseSensitive=true')

and is false per default.

It does not seem to be possible to enable it globally in $SPARK_HOME/conf/spark-defaults.conf with

spark.sql.caseSensitive: True

though. Is that intended or is there some other file to set sql options?

Also in the source it is stated that it is highly discouraged to enable this at all. What is the rationale behind that advice?

回答1:

As it turns out setting

spark.sql.caseSensitive: True

in $SPARK_HOME/conf/spark-defaults.conf DOES work after all. It just has to be done in the configuration of the Spark driver as well, not the master or workers. Apparently I forgot that when I last tried.

回答2:

Try sqlContext.sql("set spark.sql.caseSensitive=true") in your Python code, which worked for me.

来源：https://stackoverflow.com/questions/42946104/enable-case-sensitivity-for-spark-sql-globally