Spark Scala list folders in directory

Question

I want to list all folders within a hdfs directory using Scala/Spark. In Hadoop I can do this by using the command: hadoop fs -ls hdfs://sandbox.hortonworks.com/demo/<

Answer 1

Azure Blog Storage is mapped to a HDFS location, so all the Hadoop Operations

On Azure Portal, go to Storage Account, you will find following details:

• Storage account

• Key -

• Container -

• Path pattern – /users/accountsdata/

• Date format – yyyy-mm-dd

• Event serialization format – json

• Format – line separated

Path Pattern here is the HDFS path, you can login/putty to the Hadoop Edge Node and do:

hadoop fs -ls /users/accountsdata 

Above command will list all the files. In Scala you can use

import scala.sys.process._ 

val lsResult = Seq("hadoop","fs","-ls","/users/accountsdata/").!!