Bluemix spark-submit — How to secure credentials needed by my Scala jar

Question

I have a Spark application that I am submitting to the Bluemix Spark Cluster. It reads from a DASHDB database and writes the results to Cloudant. The code accesses the DASHDB using both Spark and JDBC.

The userid & password for the DASHDB database are passed as arguments to the program. I can pass these parameters via spark-submit but I don't think that would be secure. In the code I need to know the credentials of the DASHDB database because I am using JDBC to connect to various tables.

I am trying to find the "Best Practices" way to pass credentials using spark-submit in a secure manner.

Thanks in advance - John

Answer 1

I think the jdbc driver will always need username and password to connect to database so that is out of question as you are in multi-tenant enviornment on bluemix.

Now about spark-submit.sh to read the arguments securely, that option is not available yet.

Thanks, Charles.

Answer 2

Based on the answer here, my preference would be to pass a properties file that has the credentials. Other tenants will not be able to read the properties file, but you will be able to read if from your spark application, e.g. as a dataframe spfrom which you can access the parameters.