How to resolve external packages with spark-shell when behind a corporate proxy?

Question

I would like to run spark-shell with a external package behind a corporate proxy. Unfortunately external packages passed via --packages option are not resolved.

Answer 1

If proxy is correctly configured on your OS, you can use the java property: java.net.useSystemProxies:

--conf "spark.driver.extraJavaOptions=-Djava.net.useSystemProxies=true"

so proxy host / port and no-proxy hosts will be configured.