Pyspark: get list of files/directories on HDFS path

Question

As per title. I\'m aware of textFile but, as the name suggests, it works only on text files. I would need to access files/directories inside a path on either HD

Answer 1

There is an easy way to do this using snakebite library

from snakebite.client import Client

hadoop_client = Client(HADOOP_HOST, HADOOP_PORT, use_trash=False)

for x in hadoop_client.ls(['/']):

...     print x