spark-shell error : No FileSystem for scheme: wasb

Question

We have HDInsight cluster in Azure running, but it doesn\'t allow to spin up edge/gateway node at the time of cluster creation. So I was creating this edge/gateway node by i

Answer 1

Hai Ning from Microsoft has written an excellent blog post on to setup wasb on an apache hadoop installation.

Here is the summary:

1. Add hadoop-azure-*.jar and azure-storage-*.jar to hadoop classpath

   1.1 Find the jars in your local installation. It's at /usr/hdp/current/hadoop-client folder on HDInsight cluster.

   1.2 Update HADOOP_CLASSPATH variable at hadoop-env.sh. Use exact jar name as java classpath doesn't support partial wildcard.

2. Update core-site.xml

            
           fs.AbstractFileSystem.wasb.Impl                           
           org.apache.hadoop.fs.azure.Wasb 
   
   
   
           fs.azure.account.key.my_blob_account_name.blob.core.windows.net 
           my_blob_account_key 
   
   
    
   
           fs.defaultFS          
           wasb://my_container_name@my_blob_account_name.blob.core.windows.net
   
   

See exact steps here: https://github.com/hning86/articles/blob/master/hadoopAndWasb.md